I'm in favor of pursuing this more if we care strongly about static builds. We will need to investigate potential changes in performance and talk to the RocksDB maintainers about their experience using musl. golang/go#13470 has shown that static linkage of glibc into a project such as ours is just not something we should do.

Well, this is green. @mberhault: thoughts?

no idea. I haven't encountered that particular bug.
Are there any known issues when linking against musl? If not, I'm happy to see this merged and re-evaluated if we do encounter problems.

We definitely need to make sure we always use musl, not only in static
builds. After all, that's what we want to benchmark against and generally
it seems weird to use glibc most of the time but not in production. Isn't a
decision taken lightly as far as I'm concerned.

On Mon, Dec 7, 2015, 17:42 marc notifications@github.com wrote:

no idea. I haven't encountered that particular bug.
Are there any known issues when linking against musl? If not, I'm happy to
see this merged and re-evaluated if we do encounter problems.

—
Reply to this email directly or view it on GitHub
#3343 (comment)
.

@tschottdorf this patch does that, at least in the build docker container.

@mberhault looks like golang/go#13492 is the ony issue on the books right now, which seems very unlikely to affect us.

Ok, but that won't do it for linux users (make STATIC=1 release) or anything else. Not sure how important that is (will someone build their own static library?).
I didn't find anything on the internet mentioning RockDB and musl - maybe @igorcanadi can chime in here with regards to how big a change this is wrt RocksDB?

Unfortunately I'm not very familiar with musl and I'm not aware of anybody using RocksDB with musl. Maybe ask at https://www.facebook.com/groups/rocksdb.dev/?

The discussion on the go/glibc bug makes it clear that statically linking libc is considered an exotic use case. So let's step back for a minute: why is it important for us to build fully-static (as opposed to mostly-static) binaries? The Go compiler itself is distributed as a binary which dynamically links libc. Why can't we do the same?

I'm not an authority on this, but perhaps it allows us to run on any system, not just systems with the same libc of the same version or higher.

The work was first done in a80fd74.

Also not an authority on this, but from my point of view we did it purely for the deployment story. I'm also starting to think that distributing binaries which link only glibc dynamically is fine since that's what everyone else is doing (and since linking it statically is a bad idea with glibc), and since venturing out into unknown territory may not pay off.

I've added a commit here that discontinues the use of static builds. If someone does try to build a static binary, we'll try to use musl (since glibc is known bad), but usual workflows (read: tests on AWS) will use dynamic linking.

PTAL

cc @mberhault

After some additional digging, I've backed out the "stop statically linking" commit - see the latest commit message for details. Also, I've removed the netgo build tag, since there's no benefit to using netgo now that we're using musl which provides proper static linking.

Following this I was leaning towards leaving out musl for the sake of simplicity, but seeing that we link against a bunch of others it doesn't seem like such a bad idea any more. Or is it "straightforward" to link in most things statically, but not all of them?

@petermattis @bdarnell @mberhault would be good to have your opinions on this. The last commit message should provide useful context.

Do we have any evidence that the list of "standard" dynamic libs that are in that commit message are problematic?

My possibly misguided expectation is that at least /lib/x86_64-linux-gnu/libgcc_s.so.1 is problematic since it requires a gcc-like compiler to be present.

libgcc and libstdc++ look problematic: they're not present in a base tinycorelinux (i.e. boot2docker) image.

LGTM (this unblocks some of the ec2 testing work, right?). I think for our final releases we probably want to link libc dynamically, though, so we should file a separate issue to figure out how to link everything but libc statically.

Yeah, it avoids having weird workarounds to get the PGWire test running
there. Btw, if we ever want to run on busybox, we'll have to deal with
glibc.

On Wed, Dec 9, 2015 at 8:00 PM Ben Darnell notifications@github.com wrote:

libgcc and libstdc++ look problematic: they're not present in a base
tinycorelinux (i.e. boot2docker) image.

LGTM (this unblocks some of the ec2 testing work, right?). I think for our
final releases we probably want to link libc dynamically, though, so we
should file a separate issue to figure out how to link everything but libc
statically.

—
Reply to this email directly or view it on GitHub
#3343 (comment)
.

ec2 testing work is not currently blocked by this; the only test that reliably hits the original bug is TestPGWire, and that isn't being run in a static binary anywhere right now.

I don't think we'll ever need to run on an environment as minimal as the busybox docker image (which is distinct from busybox itself). In addition to libc, we also need a zoneinfo database (and presumably enough of a support structure to keep it up to date, although I guess this could just be mounted in from the host machine).

I'm inclined to merge this and file a follow-up issue as @bdarnell suggests. I'll do that after lunch tomorrow to allow time for objections.

Still pro merging. But we should file a follow-up issue to figure out how to statically link everything but glibc. @bdarnell didn't find existing resources about that on the internet. Presumably this is an issue not only for Go, so there might still be something out there (or we'll have to reach out to individuals).